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Abstract 

Randomness extraction is the process of constructing a source of randomness of high quahty 
from one or several sources of randomness of lower quality. The problem can be modeled using 
probability distributions and min-entropy to measure their quality and also by using individual 
strings and Kolmogorov complexity to measure their quality. Complexity theorists are more 
familiar with the first approach. In this paper we discuss the second approach. We present the 
connection between extractors and Kolmogorov extractors and the basic positive and negative 
results concerning Kolmogorov complexity extraction. 

1 Introduction 

Randomness is a powerful computational resource. For some problems, randomized algorithms 
are significantly faster than the best currently known deterministic algorithms. Furthermore, in 
some areas, such as cryptography, distributed computing, game theory, and machine learning, the 
use of randomness is compulsory, because some of the most basic operations simply do not have 
a deterministic implementation. 

It is not clear how to obtain the random bits that such algorithms need. While it seems 
that there are sources of genuine randomness in Nature, they produce sequences of bits with 
various biases and correlations that are not suitable for direct employment in some applications. 
For example in cryptographical protocols it is essential to use "perfect" or "close-to-perfect" 
randomness. 

It thus is important to determine whether certain attributes of randomness can be improved 
effectively, or, even better, efficiently. It is obvious that randomness cannot be created from 
nothing (e.g., from the empty string). On the other hand, it might be possible that if we already 
possess some randomness, we can produce "better" randomness, or "new" randomness. 

These general questions have been investigated in three settings: 

1 Finite probability distributions: We start with random variables Xi over {0, l}"'^ , X2 over 
{0, 1}"^, . . ., Xf over {0, 1}""*, whose distributions have min-entropy above a certain value 
that characterizes the quality of input randomness. We want a computable (or, better, 
a polynomial-time computable function) / so that f{Xi,...,Xt) is close to the uniform 
distributions (that is / produces "better" randomness), or f{Xi,...,Xt) is close to the 
uniform distributions even conditioned by some of the Xj's (that is / produces "new" 
randomness). 
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2 Finite binary strings: We start with finite binary strings xi € {0, 1}"^, X2 S {0, 1}"^, . . ., 
xt G {0,1}"', each string having some Kolmogorov complexity above a certain value that 
characterizes the quality of input randomness. We want a computable (or, better, a 
polynomial-time computable function) / so that /(xi, . . . ,xt) has close to maximum Kol- 
mogorov complexity (that is / produces "better" randomness), or /(xi, . . . ,Xt) has close to 
maximum Kolmogorov complexity even conditioned by some of the Xj's (that is / produces 
"new" randomness). 

3 Infinite binary sequences: We start with infinite binary sequences xi £ {0,1}°°, X2 G 
{0, 1}°°, . . ., Xt G {0, 1}°°, each sequence having effective Hausdorff dimension above a cer- 
tain value that characterizes the quality of input randomness. We want a Turing reduction 
/ so that f{xi, . . . ,Xt) has effective Hausdorff dimension close to 1 (that is / produces 
"better" randomness), or f{xi,...,xt) has effective Hausdorff dimension close to 1 even 
conditioned by some of the Xj's (that is / produces "new" randomness). 

The common scenario is that we start with t sources (which are distributions, or strings, or 
sequences, depending on the setting), possessing some level of randomness, from which we want to 
obtain better randomness and/or new randomness. This process is called randomness extraction. 

Setting 1 has been extensively studied and is familiar to the readers of this column. A function 
/ achieving the objective in setting 1 is called an extractor. Extractors have been instrumental in 
obtaining important results in derandomization, cryptography, data structures, and other areas. 

In this paper we discuss Kolmogorov extractors, which are the functions / achieving the ob- 
jectives in setting 2 and settting 3. The issue of Kolmogorov complexity extraction has been first 
raised for the case of infinite sequences by Reimann and Terwijn in 2003. The first explicit study 
for the case of finite strings is the paper by Fortnow, Hitchcock, A. Pavan, Vinodchandran and 
Wang [FHP^OG] (versions of the problem have been investigated earlier, for example in [ BFNV05] 
and in |VV02j ) . One reason for the late undertaking of this research line is the tight connection 
that exists between extractors and Kolmogorov extractors. However, Kolmogorov extractors have 
their own merits: they have applications in Kolmogorov complexity, algorithmic randomness, and 
other areas, and, perhaps more importantly, several general questions on randomness extraction, 
such as the amount of necessary non-uniformity or the impact of input sources not being fully 
independent, are more natural to study in the framework of Kolmogorov extractors. 

The paper is organized as follows. Section 2 contains background information on Kolmogorov 
complexity. In Section 3 we discuss Kolmogorov complexity extraction from finite strings (set- 
ting 2), and in Section 4 we discuss Kolmogorov complexity extraction from infinite sequences 
(setting 3). Section 5 presents a few applications. 

2 Basic facts on Kolmogorov complexity 

The Kolmogorov complexity of a string x is the length of the shortest effective description of x. 
There are several versions of this notion. We use here mainly the plain complexity, denoted C{x), 
and also the conditional plain complexity of a string x given a string y, denoted C{x \ y), which is 
the length of the shortest effective description of x given y. The formal definitions are as follows. 
We work over the binary alphabet {0, 1}. A string is an element of {0, 1}* and a sequence is an 
element of {0,1}°°. If x is a string, \x\ denotes its length. If x is a sequence, then x\n denotes 
the prefix of x of length n. Let M be a Turing machine that takes two input strings and outputs 
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one string. For any strings x and y, define the Kolmogorov complexity of x conditioned by y with 
respect to M, as Cm{x \ y) = min{|p| | M{p,y) = x}. There is a universal Turing machine U 
with the following property: For every machine M there is a constant cm such that for all x, 
Cu{x I y) < Cm{x I y) + cm- We fix such a universal machine U and dropping the subscript, we 
write C{x\y) instead of Cu{x \ y). We also write C{x) instead of C(a; | A) (where A is the empty 
string). The randomness rate of a string x is defined as rate(x) = ^j^- 

In this paper, the constant hidden in the O(-) notation only depends on the universal Turing 
machine. 

For all n and < n, 2^-C(i) < |{x G {0, 1}" | C(x | n) < < 2^=. 

Strings xi,X2^ ■ ■ ■ iX^ can be encoded in a self-delimiting way (i.e., an encoding from which 
each string can be retrieved) using + |x2| + . . . + + 2 log |x2| + . . . + 2 log \xk \ + 0{k) bits. 
For example, xi and X2 can be encoded as {bin{\x2\)^'ixiX2, where hin{n) is the binary encoding 
of the natural number n and, for a string u = ui . . . Um, u is the string uiui . . . UmUm (i-e., the 
string u with its bits doubled). 

The Symmetry of Information Theorem (see [ZL70] ) states that for all strings x and y, C{xy) « 
C{y) + C{y I x). More precisely: \{C{xy) - {C{x) + C{y \ x))\ < 0(logC(x) + logC(y)). In 
case the strings x and y have length n, it can be shown that \{C(xy) — (C{x) + C(y | x))\ < 

2 log n + 0(log log n). 

In Section [U we use a variant of Kolmogorov complexity, called prefix-free complexity and 
denoted K{x). The difference is that the underlying universal Turing machine U is required 
to be a prefix-free machine, i.e., the domain of C/ is a prefix-free set. It holds that for 
every string x € {0,1}", C{x) < K{x) < C{x) + O(logn). Prefix-free sets over the bi- 
nary alphabet have the following important property, called the Kraft- Chaitin inequality. Let 
{ni, n2, . . . , rifc, . . .} be a sequence of positive integers. Then ^ 2~"'* < 1 iff there exists a prefix- 
free set A = {xi,X2, • • • , Xk, ■ ■ •} with \xi\ = Ui, for all i. Moreover, if the sequence of lengths is 
computably enumerable (i.e., there is some computable / such that f{i) = Ui for all i) and the 
inequality holds, then A is computably enumerable. 

All the Kolmogorov extractors in this paper are ensembles of functions / = {fn)n<=N of type 
fn '■ ({0, !}"■)* {0, . The parameter t is a constant and gives the number of input sources. 

In this survey we focus on the cases t = 1 and t = 2. Also note that we only consider the situation 
when all the sources have the same length. For readability, we usually drop the subscript and the 
expression "ensemble / : {0, 1}" — >• {0, 1}™-" is a substitute for "ensemble / = (/n)ngNj where for 
every n, /„ : {0, 1}" {0, and similarly for the case of more sources. 

3 The finite case 

3.1 Kolmogorov extraction from one string 

We first consider Kolmogorov extraction when the source consists of a single binary string x that 
contains some complexity. For concreteness, think of the case when C{x) > an, where n = \x\ 
and (7 is a positive constant. If a is the only information that the extractor has about the source, 
then Kolmogorov extraction is impossible, as one can see from the following simple observation. 

Proposition 3.1 Let f : {0,1}" {0,1}™" be a uniformly computable ensemble of functions. 
Then, for every n, there exists a string x in {0, 1}" with C{x) >n — m and C{f{x) \ n) = 0(1). 
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Proof. Let z be the most popular string in Image(/({0, 1}")) (i.e., with the largest number of 
preimages), with ties broken in some canonical way. Since the above is a full description of z, 
C{z I n) = 0(1). The string z has at least 2""*" preimages and, therefore, there exists a string x 
in the preimage set of z with C{x) > n — m. | 

In particular, if m < an and a < 1/2, there exists a string x G {0, 1}" with C{x) > an and 
C{f{x)\n) = 0{l). 

Kolmogorov extraction may be possible if the extractor possesses additional information about 
the source x. We call this advice about the source. The basic case is when the extractor knows 
C{x). Then, one can construct x*, a shortest description of x. In other words, C{x* \ x) < 
logC(x) + 0(l) < log n + 0(1), and it is easy to see that C{x*) > \x* \ -0(1). Thus, with at most 
logn + 0(1) bits of advice about the source x, one can essentially extract all the randomness in 
the source. Buhrman, Fortnow, Newman and Vereshchagin jBFNVOS] have shown how to extract 
in polynomial time almost all the randomness in the source with O(logn) advice about the source. 



Fortnow et al. [FHP^OG] have shown that with a constant number of advice bits about the 
source, one can increase the randomness rate to arbitrarily close to 1. Moreover, their Kolmogorov 
extractor runs in polynomial time. 

Theorem 3.2 r (FHP+0 6]^ For any rational a > 0, e > 0, there exists a polynomial-time 
computable function f and a constant k such that for any x with rate(2;) > a, it holds that 
rate(/(x, Ox)) > 1 — e for some string Ox of length k. The length of f{x, ax) is at least C\x\, for 
a constant C that only depends on a and e. 

A sketch of the proof is given in Section 13.21 after we present the relation between extractors 
and Kolmogorov extractors. In the opposite direction, Vereshchagin and Vyugin [VV02| show the 
limitations of what can be extracted with a bounded quantity of advice. To state their result, let 
us fix n = length of the source, k = number of bits of advice that is allowed, and m = the number 
of extracted bits. Let K = 2'"'+^ - 1. 

Theorem 3.3 ( |VV02] ) There exists a string x G {0,1}" with C{x) > n - Klog(2™ + 1) ^ 
n — Km such that any string z € {0, 1}™ with C{z \ x) < k has complexity C{z) < k -\- logn + 
log m + 0(log log n, log log m). 

In other words, any string z that is effectively obtained from x with k bits of advice, has in 
fact unconditional complexity ~ k. 

Proof. For each x E {0, 1}", let Range(3;) = {z G {0, 1}™" \ C{z \ x) < k}. Similarly to the proof 
of Proposition 13. 11 the idea is to produce a set of strings in {0, 1}*" that is "popular," in the sense 
that is equal to Range(x), for many x € {0, 1}" (we refer to these sets as Ranges). Let T = 2™ + l. 
In a dovetailing manner, we run U{p,x) for all x G {0, l}", and all p G {0, l}-'^. We call this an 
enumeration procedure. Note that if U{p,x) halts, it outputs a string in Range(x). In step 1, we 
run this enumeration till it produces a string zi that belongs to at least 2"/T Ranges. There may 
be no such zi and we deal with this situation later. We mark with (1) all these Ranges. In step 
2, we resume the enumeration procedure till it produces a string Z2 different from zi that belongs 
to at least a fraction l/T of the Ranges marked (1). We re-mark this ranges with (2). In general, 
at step i, we run the enumeration till it produces a string Zi that is different from the already 
produced strings and that belongs to at least a fraction of l/T of the Ranges marked {i — 1) at the 
previous step. We re-mark these Ranges with (i). We continue this operation till either (a) we 
have completed K steps and have produced K strings zi, . . . ,zk G {0, 1}"^, or (b) at some step 
i, the enumeration fails to find Zi. In case (a), there are at least 2"/T^ Ranges that are equal 
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to {zi, . . . , zk}- In case (b), there are 2"/r* ^ that have {zi, . . . , Zi-i} as a subset. In addition, 
for every z G {0,1}™ — {zi, . . . , the set {zi, . . . , Zi-i, z} is a subset of less than 2"/T* 

ranges. It means that {zi, . . .,Zi_i} is equal to at least - 2™ • 2"/r* = 2"/r* Ranges. 

Consequently, the procedure produces a set {zi, Z2, ■ ■ ■ , Zs}, s < K, that is equal to Range(x) 
for at least = (-^_|_2m)y strings x E {0,1}"'. One of these strings x must have Kolmogorov 
complexity C{x) > n — log(2'" + 1). Each string Zi produced by the procedure can be described 
by i < by n, by m, and by k. We represent i on exactly k + 1 bits, and this will also describe 
k. Thus, C(zj) < A: + logn + logm + 0(loglogn,loglogm). I 

Vereshchagin and Vyugin's result explains why the Kolmogorov extractor in Theorem 13 . 21 does 
not achieve rate 1. Theorem 13.31 implies that if a single-source Kolmogorov extractor increases 
the rate from cr to 1 — e using k bits of advice, then e = r2(^^) (provided that the output length 
m is a constant fraction of n). 

3.2 Kolmogorov extraction from two strings 

We recall that a Kolmogorov extractor with two sources is an ensemble of functions of the type 
/ : {0,1}"" X {0,1}" — 7> {0,1}™. The quality of the two sources is given by their Kolmogorov 
complexity and by their degree of dependency. The dependency of two strings is the amount of 
information one string has about the other string. 

Definition 3.4 (Dependency) For x € {0, 1}", y € {0, 1}"", the dependency of x and y is given 
by 

dep(x, y) = max{C(x \ n) — C{x \ y),C{y \ n) — C{y \ x)}. 

There are in the literature several variations of the above definition. They all differ by at most 
an O(logn) additive term. For example, one may prefer C{x) + C{y) — C{xy) as a value that 
captures the dependency of X and y. It holds that \{C{x) + C{y) — C{xy)) — dep{x,y)\ =0(logn). 
Definition 13.41 tends to produce sharper statements. 

The class of sources from which we extract is characterized by two parameters: k = the 
minimum Kolmogorov complexity that each input string has, and a = the maximum dependency 
of the input strings. Accordingly, for positive integers k and a, we let 

Sk,a = {{x, y) G {0, 1}" X {0, 1}" I C{x \n)>k, C{y \n)>k, dep(x, y) < a}. 

In other words, Sk^a consists of those pairs of input sources that have complexity at least k and 
dependency at most a. 

Definition 3.5 (Kolmogorov extractor) An ensemble of functions f : {0, 1}" x {0, 1}" — >■ 

{0, 1}™ is a {k,a,d) Kolmogorov extractor if for every (x,y) G S^^a; C{f{x,y) \ n) > m — d. 

3.2.1 The curse of dependency: limitations of Kolmogorov extractors with two 
sources 

As we have discussed above, we would like to have a computable function / : {0, 1}" x {0, 1}" — >■ 
{0, 1}™ such that for all {x,y) G Sk,a, C{f{x,y)) ^ m. For a string z, we define its randomness 
deficiency to be \z\ — C{z) and thus we would like the randomness deficiency of f{x,y) to be 
~ 0. However, we will see that this is impossible. We observe that no computable function / as 
above can guarantee that for all {x,y) G Sk^a the randomness deficiency of f{x,y) is less than 
a — O(loga) + 0(1), even for a large value of k. 
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Theorem 3.6 ( [ZimlOb| ) There is no uniformly computable ensemble of functions f : {0, l}*^ x 
{0,1}" {0, 1}*^, such that for all {x,y) E Sk,a, the randomness deficiency{f{x,y)) < a — 
O(loga). 

The above holds for all k < n — a and all m > a (ignoring O(logn) additive terms). 

Proof. Let / : {0, l}" x {0, 1}" — )• {0, 1}™ be a uniformly computable ensemble of functions. We 
look at prefixes of length a of strings in the image of /. Let z be the most popular prefix of 
length a of strings in the image of /. Note that C{z j n) = 0(1). There are > 2^"~" pairs {x,y) 
with f{x,y)\a = z. There is a pair {x,y) as above with C{xy \ n) > 2n — a. It follows that 
{x,y) G Sn-a,a (ignoring O(logn) terms). Since f{x,y) = zw with \w\ = m — a, it follows that 
C{f{x,y)\n) < m — a + 2 log a + 0(1). In other words, the randomness deficiency of /(x, y) is at 
least Q — 21oga — 0(1). I 



3.2.2 Extractors vs. Kolmogorov extractors 

Positive results (within the limitations shown in Theorem 13. 6p regarding Kolmogorov complex- 
ity extraction can be obtained by exploiting the relation between extractors and Kolmogorov 
extractors. 

In the case of extractors, sources are modeled by random variables taking values over 
{0, 1}". Sometimes, such a random variable is indentified with the distribution of its output. 
The min-entropy of a distribution X over {0,1}", denoted Hoo{X), is given by Hoo{X) = 

min I log pj.oh(x—a) I ^ ^ {0, 1}", Prob(X = a) ^ o|. Thus if X has min-entropy > k, then 
for all a in the range of X, Prob(X = a) < 1/2'^. A distribution X over {0, 1}" with min-entropy 
k is called an (n, A;)-source. For each n G N, let [/„ denote the uniform distribution over {0, 1}". 
The min-entropy of a source is a good indicator of the quality of its randomness. Note that if 
Hoo{X) = n, then X = Un, and thus X is "perfectly" random. Smaller values of min-entropy 
indicate defective sources (the smaller the min-entropy is, the more defective the source is). 

For A C {0,1}", we denote ^x(^) = Prob(X G A). Let X, Y be two sources 
over {0,1}". The distance between two distributions X and Y over {0,1}" is |X — y| = 
maxAc{o,iyAl^x{A)- fiY{A)\. It is easy to show that = {l/'2)^ae{o,i}" l/'*x(a) -/"y (a)| = 

Yla■|lx{a)>^lY{a) l^ X [o) — fly {o) . The distributions X and Y are e-close if |X — y[ < e. The following 
facts are helpful. 

Lemma 3.7 ( jGUVOQQ Let D be a distribution over {0, 1}" and let HEAVY^^j = {a G {0, 1}" | 
Hoia) > t2-'=}. 

(1) If D is e-close to a distribution with min-entropy k then ^£)(HEAVYfc f) < 1/t + e. 

(2) Suppose that for every set S C {0,1}" of size K, ^d{S) < e. Then D is e-close to a 
distribution with min-entropy at least log(i^/e). 

Proof. (1) Let D' be a distribution with min-entropy k such that \D — D'\ < e. Since 1 > 
/iZ)(HEAVYk,t) > |HEAVYk,t| • ^2"'=, we have |HEAVYk,t| < 2''/t. Then ^^/(HEAVYk.t) < 
|HEAVYk,t| -2-^ < 1/t and, therefore, /iD(HEAVYk,t) < /xd' (HEAVYk,t ) + e<l/t + e. ' | 

(2) Let xi,X2, . . . , xat be an ordering of {0, 1}" such that ij-d{xi) > ij-d{x2) > ■ ■ ■ > ^d{xn)- 
Let 2^^ = {iJLD{xi)+^i.D{x2) + - ■ ■+fJ-D{xK))/K (the average of the heaviest K elements). Note that 
each of the elements xk+i, • • • , Xn has mass at most 2~'. Also, since ^ioixi) + . . . + ^J'D{xK) < e, 
we have £ > \og{K/e). Consider the distribution D' that assigns mass 2~^ to each of xi, . . . ,xk 
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and is the same as D on the elements Xk+i, . . . , xat. Then D' has min-entropy d. > \og{K/e) and 

It turns out that Kolmogorov extractors are roughly equivalent to almost extractors, which 
are in general weaker than extractors (two-source extractors is what we obtain if we take c? = 
in the next definition). 

Definition 3.8 (Almost extractor) An ensemble of functions f : {0, l}" x {0,1}" — > {0,1}"* 
is a {k,e,d) almost extractor if for all independent random variables X and Y over {0, 1}" with 
Hoo{X) > k and H^iY) > k, the random variable f{X, Y) over {0, 1}™ is e-close to a distribution 
D on {0, 1}™ having H^{D) > m - d. 

A very useful result of Chor and Goldreich |CG88] states that, in the above definition, it is enough 
to restrict the requirement to all random variables having a flat distribution, i.e., a distribution 
that assigns equally the probability mass to the elements of a set of size 2^. 

The connection between almost extractors and Kolmogorov extractors is most easily under- 
stood by looking at their combinatorial characterizations. The relevant combinatorial object is 
that of a balanced table. The approach is to view a function / : {0, 1}" x {0, 1}" {0, 1}"* as a 
table with rows in [A^] , columns in [A^] , and colored with colors from [M] , where = 2" , M = 2™ 
and we identify {0, 1}" with [N] and {0, 1}™ with [M]. For a set of colors U C [M], an [/-cell 
is a cell in the table whose color is in ^7. A rectangle is the restriction of / to a set of the form 
Bi X i?2) where Bi C [iV],i?2 C [N]. The balancing property requires that in all rectangles of 
size 2'^-by-2'', all colors appear approximately the same number of times. Depending on how 
we quantify "approximately," we obtain different types of balanced tables. Also, sometimes, we 
require the balancing property to hold not for every individual color a € [M], but for each set of 
colors U C [M] of a given size. 

To get an intuition on why balanced tables are relevant for randomness extraction, it is easier 
to consider the case of Kolmogorov extractors. To make matters concrete, suppose we are shooting 
for a Kolmogorov extractor with complexity parameter k and dependency parameter a. If the 
[A'^]-by-[A^] table / colored with colors in [M] is not balanced, then there is an element in the 
range of / that has many preimages. Arguing as in the proof of Proposition 13.11 this implies 
that / is not a Kolmogorov extractor. For the other direction, let us fix {x,y) € S^^a- Let 
B^ = {u £ {0,1}" I C{u I n) < C{x I n)} and By = {v e {0,1}" | C{v \ n) < c(y \ n)}. 
Bx X By forms a rectangle of size ~ 

2C{x\n) ^ 2C(yl"), and this is ^ 2^ X 2^ or larger (because 
C{x I n) > k,C{y \ n) > k). Suppose that the table / satisfies the following balancing property: 
Each color from [M] appears in the rectangle B^ x By a fraction of at most c/M times, where c 
is a constant. Clearly, (x, y) is a cell in Bx x By and, therefore, the color z = f{x, y) appears at 
most (c/M) • 2C(^I")+C(?'I") = 2C(^l")+C(yl")-'^+0(i) times in Bx x By. If C{x \ n) and C{y \ n) 
are given, one can effectively enumerate the elements of B^ x By. Then the string xy can be 
described by z, by C(x | n) and C(y | n), by the rank r of the cell {x,y) in an enumeration 
of the z-colored cells in Bx x By, and by the table /. Thus, C{xy \ n) < C{z \ n) + C{C{x \ 
n)) + C{C{y I n)) + C{r \ n) + (7(table [ n) + 0(log n). C{C{x \ n)) and C{C{y \ n)) are 0(log n), 
and, since the table is computed from n (because the ensemble / is uniformly computable), 
C(table I n) = 0(1). By the above estimation, C(r | n) < C{x \ n) + C{y \ n) — m + 0{1). 
We obtain C{xy \ n) < C{z \ n) + C{x \ n) + C{y \ n) — m + O(logn). On the other hand, 
from the dependency property of x and y, C{xy \ n) > C{x \ n) + C{y \ n) — a. It follows that 
C{z \ n) > m — a — O(logn), which is the desired conclusion. With a more elaborate argument, 
we can get 0(1) instead of O(logri). Since we need the above to be true for every (x, y) € Sk,a-, we 
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require that the above balancing property holds for all rectangles of size 2^ x 2^^, or larger. In fact 
it is enough to require the balancing property to hold for all rectangles of size 2^ x 2^ (because if 
there exists a larger unbalanced rectangle, then there is also a 2^= X 2'= unbalanced rectangle). 

After this motivating discussion, we pursue with the combinatorial characterization of almost 
extractors and of Kolmogorov extractors. 

Proposition 3.9 (Combinatorial characterization of almost extractors) Let f : 

{0, 1}" X {0, 1}" — )• {0, 1}™ be an ensemble of functions. 

(1) If f is a {k,€,d) almost extractor, then for every rectangle Bi x B2 ^ [N] x [N] of size 
2^ X 2^^, and for any set of colors U C [M], 

\{U-ceUs %n Bi x ^2}! ^ \U\ d. 
\Bi X S2I - M ' 

(2) Suppose that for every rectangle Bi x B2 Q [N\ x [N] of size 2^ x 2^ , for any set of colors 
[/C [M] with \U\ =e-M ■2-'^, 

\{U-cells in Bi x ^2}! ^ 1^1 d, 
\Bi X B2I - M ' 

Then f is a {k, 2e, d) almost extractor. 

Proof. (1). Let X and Y be two independent random variables that are flat on Bi, respectively B2. 
Since X and Y have min-entropy k, f{X, Y) is e-close to a distribution D with min-entropy at least 
m — d. We have Hd{U) < \U\ ■ 2^™+'^ and the conclusion follows because f^f(x,Y){U) < 1^d{U) + e 
A iTT\ |{C/-cells in _Bi XB2I} ■ 

(2) Let X and Y be independent random variables that have flat distributions over {0, 1}" 
with min-entropy k. Let Bi be the support of X and B2 be the support of Y. Then ^f(^x,Y){U) = 
\{U cellj^in^BixB2}| ^ M . _|_ £ < 2e (the first equality holds because X and Y are flat, and the 
second and third inequalities follow from the hypothesis). Then, by Lemma 13.71 (2), f{X^Y) is 
2e-close to a distribution with min-entropy equal to log(|C/|/e) = m — d. | 



Proposition 3.10 (Combinatorial characterization of Kolmogorov extractors) Let f : 

{0, l}" X {0, l}" — > {0, l}'*" be an ensemble of functions. 

(1) If f is a {k, a ^d)- Kolmogorov extractor, then for any rectangle BiX B2 ^ [N] x [N] of size 
2^' X 2*=', where k' = k + a, for any set of colors U C[M], with size \U\ = 2"° • M • 2"('^+'^(i)), it 
holds that 

\{U-cells in Bi x ^2}! ^ IJA ^d+Oii) 
\Bi xB2\ - M ' 

(2) Suppose that there exists a constant d such that for all rectangles Bi x B2 of size 2^^ x 2^, 
for any U C [M] and for some e computable from n, it holds that 

\{U-cells in Bi x ^2}! < M . + e 
\Bi X B2I - M ' ^' 

Then f is a {k',a,a + 2d + 1) Kolmogorov extractor, where k' = k + logn + O(loglogn), and 
a = log(l/e) + 1. 
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Proof. (1) Suppose there exist Bi,B2,U violating the conclusion. Specifically we assume: Bi C 

[N],\Bi\ = B2 C [N],\B2\ = 2''', where k' = k + a, U C [M], \U\ = 2™-"-<^-^i and 

'"^IbixB^^^^^ ^ ^ ' 2~"+'^i, for appropriate choices of the constants. We construct the first 
(in some canonical sense) triplet (Bi, B2,U) satisfying the above relations. Note that for every 
zGU,C{z\n)<m — a — d — ci + 0(1) < m — a — for a sufficiently large ci. 

We estimate the number of elements of Bi x B2 that are not good for extraction, i.e., the size 
of Bi X B2 — Sk^a- Bi X B2 — Sk,a is contained in the union of BADi, BAD2, and BAD3, where 
BADi = {{x,y) € Bi X B2 \ c(x \ n) < k}, BAD2 = {{x,y) e Bi x B2 \ C{y \ n) < k} and 
BAD3 = {(x,y) € Si X I C{x I n) - C{x \y)> aoi C{y \ n) - C{y \ x) > a). Clearly, |BADi| 
and IBAD2I are each bounded by 2^'^'^ . Regarding BAD3, note that if C{x \ n) — C{x [ y) > a, 
then C{x \ y) < C{x \ n) — a < k' — a + 0(1) (because, conditioned by n, x can be described 
by its rank in a canonical enumeration of Bi). Similarly, if C(y | n) — C{y \ x) > a, then 
C{y \ x) < k' - a + 0(1). It follows that IBAD3I < 2 • 22'='-°+0(i). Thus, \Bi x B2 - < 
IBAD1I + IBAD2I + IBAD3I < 2'=+'='+2^+'=' + 2-22fc'-°+'^(i) < 4.22fc'-°+'=i, for a sufficiently large ci. 
Since the number of ?7-cells in Bi x B2 > ^ - 2^^ -a+ci^ there exists a pair (x, y) G Bi x B2 Ci S^^a 
such that f{x,y) £ U. Let z = f{x,y). It follows that z £ U and C{z \ n) > m — a — d, 
contradiction. | 

(2) Fix {x,y) € S^'^a- Let z = f{x,y) and let t = a + 2d + 1. For the sake of contradiction, 
suppose that C{z \ n) < m — t. Let = C{x \ n) > k' and ty = C{y \ n) > k'. Let 
B^ = {ue {0, 1}" I C{u I n) < t^} and By = {v e {0, 1}" | C{v \ n) < ty}. Note that 2*--'^(i) < 
\B^\ < 2*-+i and 2*!'-^(i) < \By\ < 2*^+^ We take U = {u e {0,1}"^ \ C{u \ n) < m - t}. We 
have \U\/M • 2*^ + e < (2™-"-2'i-i/2'») . 2*^ + 2~°'"'^-^ = 2-"-^^. We say that a column v G [A^] 
is bad if the number of [/-cells in B^ x {v} is > 2*^"""*^. The number of bad columns is < 2^ 
(otherwise the hypothesis would be violated by the rectangle formed with B^ and the set of bad 
columns). Also, the set of bad columns can be enumerated if n and tx are given. It follows 
that if is a bad column, then C{v \ n) < k + logt^ + 2 log log + 0(1) < k + 21ogn. Since 
C{y \ n) > k', y is a good column. Therefore, the number of [/-cells in B^ x {y} is < 2*^~"~'^. By 
our assumption, {x,y) is an [/-cell in B^ x {y}. So, the string x can be described by: y, rank of 
(x, y) in an enumeration of [/-cells in Bx x {y}, tx and d. We write the rank on exactly tx — a — d 
bits and this also provides tx- It follows that C{x \ y) < tx — a — d + logd + 2 log logd + 0(1). On 
the other hand, since dep(x,y) < a, C{x \ y) > tx — a. It follows that d < log (i+2 log log (i+0(l), 
contradiction (if d is large enough). | 

Combining the combinatorial characterizations of almost extractors and of Kolmogorov ex- 
tractors, we obtain the following theorem. 

Theorem 3.11 (Equivalence of almost extractors and Kolmogorov extractors) Let f : 

{0, !}"■ X {0, !}"■ — > {0, 1}*" be an ensemble of functions. 

(1) (implicit in fFH^O^) If f is a {k, e, d) almost extractor, then f is a {k' , a,a + 2d + 1) 
Kolmogorov extractor, where k' = k + log n + 0(log log n) and a = log(l/e) + d+1. 

(2) ( ]HPV09^ ) If f is a {k,a,d) Kolmogorov extractor, then f is a {k',e,d') almost extractor, 
where k' = k + a, e = 2 ■ 2'"' and d' = d + 0(1). 

In brief, any almost extractor is a Kolmogorov extractor with a small increase in the min-entropy 
parameter, and vice-versa. In the correspondence between the two notions, the dependency 
parameter of the Kolmogorov extractor and the error parameter of the almost extractor are 
related hy a ^ log 1/e. 
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Thus, we can take any two-source extractor (recall that any two-source extractor is an almost 
extractor with the randomness deficieny parameter d = 0), and immediately conclude that it is 
also a Kolmogorov extractor. Dodis and Oliveira jDU03] showed the existence of computable 
(fc, e) two-source extractors for any k > logn + 2 log 1/e, with output length m = 2k — 2 log 1/e. 
In applications, we typically need polynomial-time computable procedures. If we focus on the 
min-entropy parameter, the currently best polynomial-time two-source extractors are due to Bour- 
gain |Bou05j . which has k = 0.4999n and m = ^}{n), and to Raz |Raz05j . in which one source needs 
to have min-entropy > 0.5n and, the second one only need to have min-entropy polylog(n). Kalai, 
Li, and Rao |KLR09j have used a hardness assumption to construct a polynomial-time two-source 
extractor for min-entropy 5n (for both sources, and constant 5) and m = n^^^\ The hardness 
assumption is the existence of one-way permutations with certain parameters. For sources with 
min-entropy > 0.5n, Shaltiel |Sha06] . has constructed a polynomial-time two-source extractor 
with k = (1/2 + 6n), e = 2~^°s and m = 2k — clog(l/e), where c is a constant that depends 
on 5. Rao |Rao08j has constructed a polynomial-time computable (/c, e, d) almost extractor for 
k = 5n, d = poly(l/(5, 1/e) and m = 0{5n). By Theorem l3.1H all these results lead to Kolmogorov 
extractors with the corresponding parameters. 

Radhakrishnan and Ta-Shma [RTSOO] showed that any two-source extractor must suffer an 
entropy loss of 2 log 1/e. Thus, any two-source extractor E : {0,1}" x {0,1}" — ?> {0,1}*", with 
parameters {k,e), must have output length m < 2k — 2 log 1/e. When we view £^ as a (A: + 
O(logn), a, d) Kolmogorov extractor, via Theorem 13. IH the dependency parameter a is ~ log 1/e. 
Recall that the randomness deficiency of a Kolmogorov extractor is at least a, which, in other 
words, means that C{E{x,y)) < m — a. Thus, at best, we obtain that for any {x,y) G Sk^a, 
C{E{x,y)) ~ 2fc — 3a. In fact we can hope that there exists an extractor E with C{E{x,y)) = 
2k— a because x and y have each k bits of randomness, of which they share a bits. For the stronger 
type of extraction in which we require that E{x, y) has maximum possible Kolmogorov complexity 
even conditioned by any one of the input strings, we should aim for C{E{x,y) | x) « A; — a and 
C{E{x,y) \y) ^ k-a. 

The latter optimal settings of parameters have been obtained for computable (but not 
polynomial-time computable) Kolmogorov extractors by Zimand in |Zim09] . and in the stronger 
form in |ZimlOb| . 

Theorem 3.12 r |ZimlOb] ) Let k{n) and a{n) he integers computable from n such that n > 
k{n) > a{n) + 71ogn + 0(l). There exists a computable ensemble of functions E : {0,1}" x 
{0, 1}" {0, 1}™, where m = k{n) — 7 log n such that for all {x,y) G Sk^a^ holds that C{E{x,y) \ 
x) = m — a{n) — 0(1) and C{E{x, y) \ y) = m — a{n) — 0(1). 

Proof sketch. As it is usually the case for constructions that achieve optimal parameters, we use 
the probabilistic method. The trick is to conceive the right type of balancing property that leads 
to the desired conclusion and that is satisfied by a random function. In our case, the balancing 
property, which we call rainbow balancing, is somewhat complicated. 

Rainbow balanced tables. The novelty is that unlike the tables in Proposition 13.91 and Proposi- 
tion [STOl where the balancing property refers to a single color per rectangle, now we require the 
balancing with respect to a different color for each column in the rectangle (and, separately, for 
each row). The table is of the form E : [N] x [N] — > [M]. We fix a parameter D, which eventually 
will be taken to be -D ~ 2"("-). Let Ad be the collection of all sets of colors A C [M], with size 
1^1 « M/D. Let i?i X i?2 ^ [N] x [A^] be a rectangle of size K x K. We label the columns in B2 
as B2 = {vi < V2 < . . . < vk}- Let A = {Ai,A2, . . . , Ak) be a K-tuple with each Ai G Ad- In 
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Table 1: Rainbow-balanced table. For each column v in B2, we choose a set of colors C [AI] of size ~ M/D, 
and we require that it does not appear more than a fraction of 2/Z) times in Bi x {u}. This should hold for all 
rectangles of size K x K and for all choices of Ay, and also if we switch the roles of columns and rows. 



other words, for each column Vi we fix a set of colors Ai. We say that a cell {u,Vi) in Bi x B2 is 
properly colored with respect to B2 and A if E{u,Vi) S ^j. Since A^ C [M] and \Ai\ ^ M/D, if 
£^ is random, we expect the fraction of cells that are properly colored with respect to B2 and A 
to be ~ 1/D. Similarly, we define the notion of a properly colored cell with respect to Bi and a 
iC-tuple A' = {A[,A2,..., A'^). Finally, we say that the [iV]-by-[iV] table E colored with colors 
from[M] is {K, L')-rainbow balanced if for all rectangles Bi x B2 of size K x K, for all iT-tuples 
A G {Ad)^ and A' G [Ad)^ , the fraction of cells in Bi x B2 that are properly colored with 
respect to B2 and A (and respectively, with respect to Bi and A') is at most 2/D. 

A standard probabilistic analysis shows that a random table E : [N] x [N] — )• [M] is {K, D)- 
rainbow balanced, provided M < K and D < K (in the latter inequalities we have omitted some 
small factors). 

We next present the construction. For readability, we hide some annoying small factors and 
therefore some of the parameters in our presentation are slightly imprecise. We take d = a{n) + 
clogn, for a constant c that will be fixed later, D = 2^, and K = 2'^(") (with a more careful 
analysis, we can take d = a{n) + 0(1)). The probabilistic argument shows that there exists a 
(K, D) rainbow balanced table. By brute-force we can effectively construct a {K, D)-rainbow 
balanced table E for every n. Fix (x,y) G Sj^(^n),a{n) and let z = E{x,y). For the sake of 
contradiction suppose that C{z \ y) < m — d. For each v, let Ay = {w & [M\ \C{w \v) < m — d}. 
It holds that Ay G Ad for all v. Let B^ = {u ^ [N] \ C{u \ n) < C{x \ n)}. Let us call a column 
V had if the fraction of cells in B^ x {v} that are ^^.-colored is larger than 2 • (1/D). The number 
of bad columns is less than K, since otherwise the rainbow balancing property of E would be 
violated. We infer that if u is a bad column, then C{v) < k{n). Since C{y) > k(n), it follows 
that y is a good column. Therefore the fraction of cells in the B^ x {y} strip of the table E that 
have a color in Ay is at most 2 • (l/D). Since {x,y) is one of these cells, it follows that, given 
y, X can be described by the rank r of {x, y) in an enumeration of the ^y-colored cells in the 
strip Bx X {y}, a description of the table E, and by O(logn) additional bits necessary for doing 
the enumeration. Since y is a good column, there are at most 2 • (l/D) • w 2"^+! • 2'^(^) 
cells in B^ x {y} that are Aj^-colored and, therefore, logr < C(x) — d + 1. From here we obtain 
that C{x I y) < C{x) — d + 1 + O(logn) = C{x) — a{n) — clogn + O(logn). Since C{x \ y) > 
C{x) — a{n), we obtain a contradiction for an appropriate choice of the constant c. Consequently 
C{z I y) > m — d = m — a{n) — clogn. Similarly, C{z \ x) > m — a(n) — clogn. With a more 
careful analysis the clogn term can be replaced with 0(1). Thus we have extracted m ~ k{n) bits 
that have Kolmogorov complexity ~ m — a(n) conditioned by x and also conditioned by y. | 

The proof of Theorem 13.21 is also based on the equivalence beteween multi-source extractors 
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and Kolmogorov extractors. 

Proof sketch of Theorem \3.2l The main tool is the polynomial-time multi-source extractor of 
Barak, Impagliazzo and Wigderson |BIW04) . which, for any it > and c > 1, uses £ = poly(l/c7, c) 
independent sources of length n, with min-entropy an, and outputs a string of length n that is 
2~'^"-close to Un- Recall that the extractor in Theorem 13.21 works with a single source x with 
randomness rate at least a. The string x is split into £ blocks xi,X2, . . . ,X£, each of length n, 
with the intention of considering each block as a source. The main issue is that no independence 
property is guaranteed for the blocks xi,...,xi and therefore the extractor E from [BIW04] 
cannot be used directly. However, one of the following cases must hold: (1) There exists xj 
with C{xj) low; in this case, since rate(x) > a, there must also exist xi with rate(xj) > o" + 7, 
for some appropriate 7; (2) the dependency of xi,...,xi is high (i.e., the number of "shared" 
random bits is high); in this case again one can argue that there exists Xi with rate(xj) > o" + 7; 
(3) the dependency oi xi, . . . ,xi is low; in this case, similarly to Theorem 13. the multi-source 
extractor E is also a Kolmogorov extractor (with £ sources) and rate(£'(xi, . . . , x^)) is close to 1. 
Thus, either Xj, in cases 1 and 2, or E{xi, . . . ,Xi), in case 3, has randomness rate higher than x. 
Iterating the procedure a constant number of times, we obtain a string with rate 1 — e. For this 
to work, we need to know, for each iteration, which one of Cases 1, 2, or 3 holds and the index i 
(for Cases 1 and 2). This constant information is given by the advice string Ux- I 

4 The infinite case 

Effective Hausdorff dimension is the standard concept that quantifies the amount of randomness 
in an infinite binary sequence. This concept is obtained by an effectivization of the (classical) 
Hausdorff dimension, and, as we discuss in Section 14.11 has an equivalent formulation in terms 
of the Kolmogorov complexity of the sequence prefixes. Namely, for each x S {0, 1}°°, dim(x) = 
lim inf ^^^^"'^ = lim inf ^^^"^^ (gee Section 14.11 for the first equality; the second equality holds 
simply because the plain and the prefix Kolmogorov complexities are within O(logn) of each 
other). 

The issue of extraction from one infinite sequence has been first raised by Reimann and Terwijn 
in 2003 (see |Rei04] ) . They asked whether for any sequence x with dim(x) = 1/2 there exists 
an effective transformation / such that dim(/(x)) > 1/2 (the value 1/2 is arbitrary; any positive 
rational number plays the same role). Formally, we identify an infinite sequence x with the set 
of strings having x as its characteristic sequence, / is a Turing reduction corresponding to some 
oracle machine M, and f{x) is the set computed by M^, i.e., the n-th bit of f{x) is 1 iff accepts 
the n-th string in the lexicographical ordering of {0, 1}*. In case halts on every input, we 
also say that /(x) is computed from x. Initially, some partial negative results have been obtained 
for transformations / with certain restrictions. Reimann and Terwijn |Rei04] have shown that 
the answer is NO if we require that / is a many-one reduction. This result has been extended 
by Nies and Reimann |NR06] to wtt-reductions. Bienvenu, Doty, and Stephan |BDS09j have 
obtained an impossibility result for the general case of Turing reductions, which, however, is valid 
only for uniform transformations. More precisely, building on the result of Nies and Reimann, 
they have shown that for all constants ci and C2, with < ci < C2 < 1, no single effective 
transformation is able to raise the dimension from ci to C2 for all sequences with dimension at 
least ci. Finally, Miller [Mil08] has fully solved the original question, by constructing a sequence 
X with dim(a;) = 1/2 such that, for any Turing reduction /, dim(/(x)) < 1/2 (or f{x) does not 
exist). We present Miller's result in Section [4. 2[ 
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4.1 Hausdorff dimension, effective Hausdorff dimension, and Kolmogorov 
complexity 

The Hausdorff dimension is a measure-tlieoretical tool used to create a distinction between sets 
that are too small to be differentiated by the usual Lebesgue measure (see, for example, Terry 
Tao's blog entry |Tao09j for an illuminating discussion). The sets of interest for us are subsets of 
[0, 1] and we restrict the definitions to this case. 

For a E {0,1}*, \a\ ([c]^'^) is the set of all binary sequences (respectively, strings) having a 
as a prefix. For V C {0, 1}*, [F] = UaeyM I^]^'^ = U^evH^'^- ^ a string r is a prefix of 
a string a, we write t <a. We use [i to denote the Lebesgue measure on {0, 1}°°, determined by 
//(H) = 2-H. 

Let A C {0, 1}°° and W Q {0, 1}*. W is an n-cover of A if all strings a in W have |fT| > n 
and A C UaewM- For s G M^o, we define H^{A) = inf {^^g^^. 2-*l'^l | W n-cover of A} and 
H^(A) = lim„_j.oo H^{A) (the limit exists). It can be shown that there exists a unique s such that 
for all t > s, H^(A) = and for all < u < s, H^{A) = oo. The (classical) Hausdorff dimension 
of A is defined as dimH(A) = inf{s > | H^'iA) = 0}. 

We illustrate the definitions with the analysis of the Cantor set. The underlying alphabet for 
this standard example is {0, 1, 2}, and we need to adapt the definitions for this setting by substitut- 
ing "2-^l'^l" with "S-^l'^l ." The Cantor set is C = {x G {0, 1, 2}°° | x contains only Os and 2 }. The 
set W = {0, 2}" is an n-cover of C. If s > 1/ log 3, H^{C) < jZxew 3"'"' = 2" • S"''", which goes to 
as n grows. Thus, dim//(C) < 1/ log 3. If s < 1/ log 3, one can check that W is an n-cover which 
yields the infimum in the definition of H^{C). Thus, in this case, H^{C) = ^^i^w 3~'^" = 2" •3"''", 
which goes to infinity as n grows. So, d\m.}{{C) > l/log3. We conclude that the Hausdorff di- 
mension of the Cantor set is l/log3. 

We return to the binary alphabet. It can be shown that H^{A) = iff 3C C {0, 1}* such that 
^f^^c 2"^^'^^' < oo and, for all x € A, 3°° a G C with x G [a]. One way to define effective Hausdorff 
dimension is to require that the set C from above is computably enumerable. 

Definition 4.1 (1) A set A C {0,1}°° has effective s- dimensional Hausdorff dimension 
(where s is a nonnegative rational number) if3C Q {0, 1}*, c.e., such that X^o-gC 2~*I'^I < oo 
and, for all x A, 3°°a G C with x G [a]. In this case we write Hf{A) = 0. 

(2) The effective Hausdorff dimension of A C {0,1}°° is diin]j{A) = inf{s > | Hl{A) = 0}. 
If A ^ {0, 1}°°, instead of dim\j{{ A}), we simply write dim)^(yl). 

One can define effective Hausdorff dimension in a slightly different manner using Solovay tests. A 
Solovay s-test is given by a c.e. set T of rational subintervals of [0, 1] such that J2ieT \^\^ < 
A set A C {0, 1}°° is covered by T if any element of A is contained in infinitely many intervals 
of T. Reimann |Rei04| has shown that dim}^(A) = inf{s > | ^ is covered by a Solovay s-test}. 
Since from now on we will be only using effective Hausdorff dimension, we abbreviate dim}^ by 
just dim. 

For us, the most relevant is an alternate characterization of effective Hausdorff dimension 
based on Kolmogorov complexity. 

Theorem 4.2 ( |Rya84| , |May02| , IStaOSj l For any x G {0, 1}°°, dim(x) = liminf 



n 



Proof, "dim(x) < liminf .^Mlli " Lgt s > liminf EiEll}!^ \Yg show that Hf{x) = 0, which implies 
dim(x) < s, from which the conclusion follows. Take C = {a {0, 1}* [ K{a) < s\a\}. Note 
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that: (a) C is c.e., (b) (3°°n) x|"n G C, (c) Zlo-eC 2"*''^' < c« (because a eC implies K{a) < s\a\ 
and therefore 2-^('^) > 2"^l'^l and ^ 2"^('^) < 1 by Kraft-Chaitin inequahty). Thus (1) in 
Definition 14.11 is satisfied. 

"fiminf "^^^^"^ < dim(3;)." Let s be such that Hl(x) = 0. We show that Uminf ^^^"^ < g_ 
We know that there exists a c.e. set C such that X^o-eC ^ ^ ^'^'^ 3°°(T G C with x S [cr]. 

For some constant c, ^o-ec ^ '^''^'^^ — Using the other direction of Kraft-Chaitin theorem, it 
follows that for all a £ C, K{a) < s\a\ + 0(1) and therefore ^ s. Consequently, 3°°n 

K(x\n) 0(i) ^ which implies liminf '^^^^"^ < s. I 

4.2 A strong impossibility result: Miller's theorem 

Theorem 4.3 ([MilOSj) T/iere exists A € {0,1}°°, mi/i dim(A) = 1/2, such that any B G 
{0, 1}°° computable from A has dim(i3) < 1/2. 

Proof. We use the notation introduced in Section [4.11 For S C {0, 1}*, we define the direct weight 
of S by DW(5) = EaG5 ^"'"'/^ and the weight of S by W{S) = inf{DW(F) | [5] C [V]}. A set V 
that achieves the infimum in the definition of W(S) is called an optimal cover of S. An optimal 
cover exists for any set S for the following reasons. If S is finite, then it is not optimal to consider 
in a cover of [S] a string r that does not have an extension in S; so there are only finitely many 
candidates for an optimal cover. If S" C {0, 1}* is an infinite set, then consider an enumeration 
{StlfgN of S, i.e., an increasing sequence of finite sets such that S = U5j. Let S°'^ be the optimal 
cover of St- The only way for a string a G 5'°'^ to not be in S^^^ is if there exists a string r ^ o" in 
S^l^. This shows that [5f ] C [S°Xi\ and that the sets S'f^ have a hmit V, with [S] C [V\. This set 
has the property that DW(y) = W(S'). So we define S°''{S) to be the set V with \S] C [V] and 
DW(y) = W(S') (if there is a tie, we pick V with the minimum measure). If S is c.e., it does not 
follow that S"^ is c.e. However, if {S'fjtgN is an effective enumeration of S and V = US'°'^, then V 
is c.e., [V] = [S°^] and for any prefix-free set P C y it holds that DW(P) < DW(5°'=) = W(5). 
A key fact is shown in the next lemma: For any c.e. 5", the measure of [5'°'^] (viewed as an infinite 
binary sequence obtained through binary expansion) has effective dimension at most 1/2. 

Lemma 4.4 If S is c.e., then dim(/i([5"^])) < 1/2. 

Proof. If S°'^ is finite, then /i([S'°'^]) is rational and thus has effective dimension 0. So assume that 
is infinite. Let w = W{S) and let V be the set from the paragraph preceding the lemma. Let 
(^)tGN be an effective enumeration of V, with Vq = 0- For an arbitrary s > 1/2, we construct a 
Solovay s-test T that covers /i([y]). Since [V] = [S°'^], this will establish the lemma. T has two 
parts, To and Ti. 

(a) If r G Vt+i - Vt, then put MVt+i]), K[Vt+i]) + 2-\^\] into Tq. 

(b) If, for some /c,n G N, fi{[Vt D {0, 1}>"]) < A; • 2"" and n{[Vt+i D {0, 1}>"]) > k ■ 2"", then 
put [fi{[Vt+i]),fi{[Vt+i]) + 2""] into Ti. 

Clearly, T = TqUTi is a c.e. set of rational intervals. Let us show that T is a Solovay s-test. First 
we analyze Tq. 

E/GTo 1^1' = EreV 2-^1^1 = En 2"- H {0, 1}" | 

= En 2(V2-^)n2-n/2 • |y H {0, 1}" | = ^„ 2'-^/2-s)n . DW(y R {0, 1}") 
< E„ 2(1/2-*)" .w = W^^ 2(V2-s)n < 



14 



where in the transition to the last hne we have used that V PI {0, 1}" is prefix-free and the above 
property of V. We move to Ti. Fix n and let k be the number of intervals of length 2~" added to 
Ti. By construction, A: • 2"" < n {0, 1}>"]). Let P C F n {0, 1}>" be a prefix-free set such 
that [P] = [Vn {0, 1}>"]. Then fi{[P]) = Y^reP 2"'"' < E.eP = 2-"/2 ^^^^ 2-I-I/2 = 

2-"/2 . DW(P). So, /t • 2-" < 2-"/2 . DW(P) < 2-"/^ . ^ ^nd thus k < 2"/^ . w_ Therefore, 
Y^i^T^ \^\' < En 2"'^^ • w ■ (2-")^ < 00. We conclude that Ti is a Solovay s-test, and so T is a 
Solovay s-test. Next, we show that T covers //([y]). Call t G V timely if only strings longer than 
r enter V after r. Let us fix a timely r, let n = |r| and let 1 be the stage when r enters V. We 
claim that there is an interval of length 2~" in T that contains When r enters V, we put 

the interval [//([^t+i]), 2""] in Tq. Let / = [^{[Vu]) , fi{[Vu] + 2""] be the last interval of 
length 2"" added to T. If /, then > /i([T4]) + 2"". By the construction of Ti, 

another interval of length 2~" is added to Ti C T after stage u, which is a contradiction. Thus, 
we conclude that for every n that is the length of a timely element of V, there is an interval of 
length 2~" in T that contains ;u([y]). Since there are infinitely many timely elements, /^([T^]) is 
covered by T. | 

Construction of set A. Let (^'e)eeN be an effective enumeration of all oracle Turing machines, 
and let 'if^lk denote the initial segment of length k of the characteristic sequence of the set 
accepted by ^'e with oracle A. 

The set A is constructed in stages so that it satisfies all requirements iie,n defined as 

Key. If is total, then {3k > n)K{'^^\k) < (1/2 + 2 • 2-")A;, 

which implies that any set computed from A has effective dimension at most 1/2. The construc- 
tion defines a sequence of conditions. A condition is a pair (cr, 5), where a G {0,1}*, S C [cr]^'^ 
is a c.e. set, and a ^ The string a will be an initial segment of A, and the set S defines 
some obstructions for A in the sense that we need to guarantee that A ^ [S"'^]. Thus, we define 
P{a,s) — ['^] ~ [5'°'^] 5 which is viewed as the set of possibilities for A according to condition (a, S) . 

At stage t, we define the condition {at, St) so that any set A in P^g.^ satisfies requirement 
i?e,n for (e, n) = t. We make sure that P(at+i,St+i) — ^(crt,St) P(at,St) empty. 

Finally, we take A to be the limit of the strings at and we also ensure that dim(A) = 1/2. 

We first list a few useful properties of conditions {a, S). 

Fact 1. If P{a,s) is not empty, then it has positive measure. This follows from a calculation 
similar to one used in Lemma |4.4[ 

Fact 2. dim(/u(P(<^,5))) < 1/2. Note that fi{P{a,s)) = 2"''^' - ^([5"°"^ n [a]<'^]) and we invoke 
Lemma 14.41 

Fact 3. If (o"i. Si), . . . , {an, Sn) are conditions such that P^^.^ n . . . fl P(o-,i,s„> positive 
measure, then there exists a condition {t,T) such that P{t,t) ^ P{ai,Si)i all 1 < i < n. 

Armed with these facts, we proceed to describe the construction. At stage t = 0, we take 
the condition (A,S'o), where Sq = {a G {0,1}* | K{a) < |<7|/2}. It can be checked that (A, So) 
is indeed a condition. Since A € P(^x,So)^ follows that dim(A) > 1/2. Since A is of course 
computable from A, the construction also guarantees that dim(A) < 1/2, and thus dim(A) = 1/2. 

At stage t + 1 = (e, n) we satisfy the requierement Re,n- First we choose 5 G N such that 
l^{P{(7t,St)) > 2^^. Let a € {0,1}* be an initial segment of the binary expansion of iJ,{P(^„^^St)) 
length m > n + b, where m is sufficiently large for what follows, and K{a) < (1/2-1- 2~"')(m — b). 
Such a string a exists because dim(/u(P^o-j 5^^)) < 1/2. For each r G {0,1}*, we define Tr = 
{v >- at,T :< ^g}- There are two cases to consider, depending on whether there exists or not 
T G {0, 1}"-^ such that St U has large measure (specifically, larger than 2 1°"*! — .a). 
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Case 1. There exists r € {0, 1}'""'' such that lJ-{P{at,StUT.r)) < (i-^-; ^(5'* UT,-) is large, that 
is there are many extensions of at that compute via ^'e the same initial segment r). Note that, 
given a and t, one can enumerate the strings satisfying the above property. Let r be the first such 
string in the enumeration. Since r is essentially described by a and a few additional bits, it follows 
that K{t) < K{a) + 2-"-{m-b) (if m is sufficiently large), and thus ir(T) < (l/2 + 2-2~")(m-6). 
On the other hand, since l^{P{at,StVJTr)) < ii{P(^„^^St)) — -^^ follows that there exists a 

string at+i € such that at+i 2 [S°'^]- We take St+i = [c^t+i]^^ H St- It can be checked that 
(c7t+i, St+i) is a valid condition with ^ P(at+i,St+i) — P(at,St)- remains to check that R^^n has 
been satisfied. By construction, at+i ^ A and r ^ "fe*^^ (because fit+i S T,-). Thus, r ^ 
Also, \T\ = m-b>n. Then, ir(^'^ | m - 6) = if(r) < (1/2 + 2 • 2-")(m - 6). 

Case There is no r as in Case 1. We satisfy -Re.m by guaranteeing that is not total. 
In Case 2, l^{P{at,StUT^)) > -O" for all r G {0, 1}^"^ Since n{P{„^^St)) < -O" + 2"™, it follows that 
/i(P^o-j — -P(o-t,StUTr>) < 2""^, i.e., the obstructions added by each have very small measure. 
There are 2"^~^ such Tj- and thus the union of obstructions added by all T^- has measure < 2"^"* • 
2"™ = 2"''. Since P^g.^ has measure > 2"'', it follows that the measure HtgIo i}^-b P{at,StUT-r) 
has positive measure. Thus, by Fact 3, there exists a condition {at+i, St+i) that extends {at, St U 
Tr) for ah r G {0, 1}™"*. Now, suppose that is total and let r = ^^\{m - h). Since < A, 
there is some p ^ Ain Tr, which implies that A G [S'fUT^] C [(S'fUT^)'"'] and hence A P(at,StUTr}^ 
contradiction. | 



4.3 Positive results regarding Kolmogorov extraction from infinite sequences 

Taking into account Miller's Theorem 14.31 one can hope for positive results only if 

(a) the Kolmogorov extractor uses at least two independent sequences, or 

(b) it uses one sequence but the randomness requirement on the output is weaker than effective 
Hausdorff dimension 1. 

We present the main results for these two situations. There is no room here for proofs; self- 
contained proofs can be found in Chapter 12 of [DHlOj . 

Regarding (a), a first observation is that it is not obvious what independence means for 
sequences. Levin [Lev84j has suggested a notion of algorithmical mutual information based on the 
corresponding concept in classical information theory. However, Levin's proposal is technically 
complicated and some basic questions remain open. For example, in Levin's setting, it is not 
clear if every sequence is dependent with itself. Finding the "right" definition of independence 
for sequences is an important open problem in algorithmical randomness theory (see [DowlOj ). 
Calude and Zimand |CZ10j have several proposals that are perhaps not tight but are natural and 
good enough for the working mathematician. In particular, a notion of independence from [CZlOj . 
which is called C-independence in [DII10| . is sufficient for Kolmogorov extraction. We say that 
sequences x and y are C-independent if C{x\n y\m) > C{x\n) + C{y\m) — 0(logn + logm), for 
all n and m. With this definition, Kolmogorov extraction is possible in situation (a). 

Theorem 4.5 ([ZimlOc]) For every rational number a > 0, there exists a Turing reduction 
(actually a truth-table reduction) f, such that for all C-independent sequences x and y, with 
dim(x) > a and dim(y) > a, it holds that dim(/(x,y)) = 1. Moreover, f is uniform in a. 

For (b), the relaxation is to require that the effective packing dimension of the output is close to 1. 
The effective packing dimension of a sequence x, denoted Dim(a;), is in many ways the dual of the 
effective Hausdorff dimension dim(x), and, analogously to Theorem l4.2t admits a characterization 
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based on Kolmogorov complexity: Dim(a;) = limsup , Fortnow et al. [FHP"'"06 show that 



it is possible to construct a sequence with packing dimension close to 1 from any sequence x with 
Dim(3;) > and a lower bound of Dim(x). 

Theorem 4.6 ( (FHP"'"0 6]) For every e > and every a > 0, there exists a Turing reduction f 
such that for every sequence x with Dim(x) > a, it holds that Dim(/(a;)) > 1 — e. Moreover, f is 
a polynomial-time computable reduction. 

Conidis (ConlO| shows that 1 — e cannot be replaced by 1 in Theorem l4.61 His result, which can 
be viewed as the analog of Miller's Theorem for effective packing dimension, shows the existence 
of a sequence x with Dim(2;) > 1/4 such that for every Turing reduction /, Dim(/(x)) < 1 (or 
f{x) is not defined). On the other hand, it is open whether from a sequence x with dim(x) > 
it is possible to effectively construct f{x) with Dim(/(2;)) = 1. 

Doty [DotOSj shows that from any sequence x with dim(x) > and a good upper bound of 
dim(x), one can construct a sequence with effective packing dimension close to 1. 

Theorem 4.7 ( |Dot08l ) For every rational (3 there exists a Turing reduction f such that for 
every sequence x with dim(x) < 13 it holds that Dim(/(x)) > 1 — e, where e = (/3/dim(x)) — 1. 

Another related result is due to Bienvenu, Doty, and Stephan |BDS09j . 

Theorem 4.8 ( |BDS09] ) For every e > 0, there exists a Turing reduction f such that for every 
sequence x, it holds that dim(/(x)) > (dim(x)/Dim(x)) — e. Thus, i/dim(x) = Dim(a;), we have 
dim(/(x)) = 1 — e. 



5 Applications 

We discuss here several applications of Kolmogorov extractors. 

(a) Hitting properties. Many technical utilizations of extractors exploit the fact that an extrac- 
tor E maps its domain almost uniformly to its range and therefore E "hits" any subset of its range 
proportionally to the density of the set. The Kolmogorov complexity spin allows the derivation of 
special properties regarding the way in which a Kolmogorov extractor hits computable subsets of 
its range. For instance, let A C {0, 1}* be a set such that A^'^ is computable by circuits of size s{n). 
Then each string z in A^^ has complexity C{z \ n) < s(n) + log [j4^"|-|-c, for some constant c. Let 
£^ be a Kolmogorov extractor such that for every (x, y) G Sk^a, C{E{x, y) \ n) > s(n)+log |^^"|+c. 
Then we deduce that E{Sk^a) does not hit A at all, i.e., for all (x,y) S Sk^a-, E{x,y) A. 

The most natural domain where Kolmogorov extractors have applications is the Kolmogorov 
complexity theory. We discuss two examples from the papers IZimlOaj and |Ziminbj . 

(b) Counting dependent strings. Given an n-bit string x and a natural number a, it is useful 
to estimate the number of n-bit strings y about which x has a bits of information, i.e., the size 
of = {y G {0, 1}" I C{y I n) - C{y \ x) > a}. The upper bound < c • 2""", for a 
constant c, is easy to derive. For finding a lower bound, there is a "normal" and simple approach 
that is best illustrated when x is random. In this case, the prefix x(l : q) of x of length a is 
also random and, therefore, if we take z to be an (n — a) long string that is random conditioned 
by x(l : a), then C{zx{l : a)) = n — O(logn), C{zx{l : a) | x(l : a)) = n — a — O(logn), and 
thus, zx{l : a) G B^a+oiiogn)- There are approximately 2"~° strings z as above, and this leads 
to a lower bound of 2^'° for |-B3;,a+o(iogn)|) which implies a lower bound of (l/poly(n))2"^" for 
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|-Ba;,«|- This method is so basic and natural that it looks hard to beat. However, using properties 
of Kolmogorov complexity extractors, we derive a better lower bound for that does not 

have the slack of l/poly(n), in case C(x) > a + O(logn) and a is computable from n (even if a 
is not computable from n, the new method gives a tighter estimation than the above "normal" 
method). Recall that there exists an extractor E that on input (x,y) G S/^^a outputs an m-bit 
string z with m m k and Kolmogorov complexity equal torn — a — 0(1) even conditioned by any 
one of the input strings. We fix x G {0, 1}" with C{x) > k. Let z be the most popular image 
of the function E restricted to {x} x {0, l}*^. Because it is distinguishable from all other strings, 
given X, z can be described with only 0(1) bits. Choosing m just slightly larger than a we arrange 
that C{z \ x) < m — a — 0{1). This implies that all the preimages of z under E restricted as above 
are are bad- for- extraction, i.e., they are not in Sk,a- Since the size of E~^{z) fl {{x} x {0,1}") 
is at least 2""™, we see that at least 2"~™ pairs {x,y) are bad-for-extraction. A pair of strings 
(x, y) is bad-for-extraction if either y has Kolmogorov complexity below k (and it is easy to find 
an upper bound on the number of such strings), or if y G B^^a- This leads to the lower bound 
> (1/C)2"— -poly(n)2". 

(c) Impossibility of independence amplification. The dependency of two strings x and y is 
another attribute (besides randomness deficiency) of randomness imperfection. Therefore, one 
would like to decrease the dependency of strings (in other words, to amplify their independence), 
i.e., one would like to have computable functions /i and /2 such that for all strings x and y satis- 
fying certain properties, dep(/i(x, y), /2(x, y)) < dep{x,y). Unfortunately, effective independence 
amplification is impossible for strings (x, y) G Sk^a and this can be easily shown using Kolmogorov 
extractors. Indeed, if for all (x,y) G S^^a, dep(/i(x, y), /2(x, y)) = f3 < a — O(loga), then, from 
/i(x,y) and /2(x,y), one could effectively produce a string z with randomness deficiency /3, and 
this contradicts the "curse of dependency" Theorem 13.61 
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